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The findings from this review do not reflect the full body of research evidence on small schools of choice. 


What is this study about? 

The study examined whether winning an admissions 
lottery to attend a small school of choice (SSC) 
in New York City improved high school gradua- 
tion rates, influenced the type of diploma students 
received, or increased the likelihood of college 
readiness. An SSC is a small, nonselective public 
high school emphasizing academic rigor, strong 
relationships between students and teachers, and 
community partnerships. 

Researchers analyzed data on more than 12,130 
students who participated in ninth-grade admissions 
lotteries for 84 SSCs in New York City (from an initial 
sample of 14,969 students in lotteries for 84 SSCs). 
The study included three cohorts of first-time ninth 
graders who participated in lotteries for SSC admis- 
sion during the 2004-05, 2005-06, or 2006-07 school 
years, and each cohort was followed for 4 years. 

Study authors measured the effect of enrolling in an 
SSC by comparing outcomes of students who won 
an admissions lottery and those who lost the same 
lottery. Students who participated in a lottery for their 
first-choice SSC but lost could enter lotteries for 
SSCs lower down their priority list. But, for the pur- 
poses of this WWC review, the WWC focuses on the 
analysis of students that uses the condition to which 
they were initially assigned (offered admission or not) 
during their first lottery as the contrast of interest. 2 

Outcome measures assessed in this impact study 
included students’ 4-year graduation rate, the type 


of high school diploma received (Local, Regents, or 
Advanced Regents), and college readiness in Eng- 
lish and math, measured using Regents English and 
Math A scores. 3 


Features of Small Schools of Choice (SSCs) 


SSCs are high schools that serve approximately 100 
students per grade, predominantly economically 
and educationally disadvantaged students of 
color. These schools are “nonselective,” meaning 
that they do not have any entrance requirements 
(e.g., no examination, interview, portfolio), and 
emphasize academic rigor, strong relationships 
between students and teachers, and community 
partnerships. 

SSCs were originally founded in 2002 by planning 
teams comprised of teachers, administrators, and 
community partners. Together, they developed the 
mission and planned the curriculum for each new 
school. 

Utilizing substantial new resources and start-up 
grants, the SSCs formed partnerships with 
organizations that have experience starting new 
schools. The staff at SSCs are primarily focused 
on academic rigor and personal relationships with 
students, which is possible given the school’s small 
organizational structure and small student body. 
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What did the study find? 

The study authors found, and the WWC confirmed, 
that an offer of admission to an SSC increased 
4-year graduation rates (in particular, increases in 
rates of receiving a Regents diploma) and college 
readiness in English, but had no effect on college 
readiness in math or other types of high school 
diplomas that students received. 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards without reservations 

The study is a well-executed randomized controlled 
trial. 
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Appendix A: Study details 

Bloom, H., & Unterman, R. (2013). Sustained progress: New findings about the effectiveness and 
operation of small public high schools of choice in New York City. New York: MDRC. 

Setting The study took place in public high schools in New York City. 

Study sample This study utilizes the natural experiment that occurs in the New York City High School Appli- 

cation Processing System (HSAPS), where, in certain situations of oversubscription to schools, 
students are randomly assigned to either be offered attendance at a small school of choice 
(SSC) or to not be offered attendance at the school. There are two components of the HSAPS 
that together produce an opportunity for a random assignment lottery to be used to compare 
the outcomes of students offered attendance to an SSC with those of students not offered 
attendance: 

(a) Students rank their preferences for up to 12 high schools. 

(b) Each school assigns priority ratings to students, based on geographic proximity and 
whether the student is “known” to the school (for example, the student contacted or visited 
the school). This produces prioritization groupings of students for each school; students in 
higher priority groupings are given priority for enrollment in the school. 

The HSAPS randomly determines the order in which it assigns students to high schools, and 
schools fill their available slots starting with students in the highest priority groupings. As long 
as the number of available seats is larger than the number of students in the highest remaining 
priority grouping, then all students in that group are assigned to the school. As the school pro- 
gressively fills seats and works down to lower priority groups, it may arrive at a point where the 
next priority group has more students in it than there are remaining seats available at the school. 
Among students in that priority group, a lottery takes place to randomly determine which stu- 
dents will be offered the available slots. In groupings where all or no students in a given grouping 
are offered a slot, no lottery takes place. Students who participate in a lottery but do not win are 
entered back into the pool for placement in SSCs lower in their preference ranking. 

This study focuses on students who participated in an SSC lottery and, in particular, their first 
SSC lottery. Although these students may win a subsequent lottery, for the purposes of this 
study, they are considered lottery losers regardless of the outcome of subsequent lotteries. 

From an initial population of 22,816 students participating in their first SSC lottery, 14,969 
students in 84 schools were included in the baseline sample. 4 The analysis sample included 
12,130 students from 84 schools. Students without follow-up data or whose lottery no longer 
had an intervention or a comparison group after excluding students with missing follow-up data 
were not included in the analysis sample. Overall, 46% of the sample students were Hispanic, 
44% were African American, 5% were White, 3% were Asian, and less than 1 % were American 
Indian. Forty-seven percent of the sample students were male, 84% were eligible for free or 
reduced-price lunch, 7% were English language learners, and 6% received special education 
services. 5 
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Intervention 

group 

First-time ninth graders who won their first SSC lottery became the intervention group, regard- 
less of whether they actually enrolled in the school. According to the authors, a “small minor- 
ity” of first-time SSC lottery winners did not enroll in an SSC (pp. 47-48; the authors did not 
quantify this amount in the report). 

Comparison 

group 

First-time ninth graders who lost their first SSC lottery became the comparison group, regard- 
less of whether they won a subsequent lottery for another SSC or enrolled in an SSC. Accord- 
ing to the authors, a “small minority” of lottery losers enrolled in an SSC (p. 47; again, the 
authors did not quantify this amount in the report). 

Outcomes and 
measurement 

Eligible outcomes included 4-year high school graduation rates, the type of diploma received 
(Local, Regents, Advanced Regents), and college readiness in English and math. College 
readiness was defined as receiving a score of 75 or higher on the Regents English and Math A 
tests. For a more detailed description of these outcome measures, see Appendix B. 

Support for 
implementation 

According to the authors, most SSCs received philanthropic and external programmatic sup- 
port during start-up. 

Reason for 
review 

This study was reviewed by the WWC in response to a request by the Institute of Education 
Sciences (IES). 
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Appendix B: Outcome measures for each domain 


Completing school 

Graduated from high school 

A binary measure indicating whether the student graduated from high school within 4 years and the sum of the 
three mutually exclusive subcategories, described below. 

Local diploma granted 

A binary measure indicating whether, upon graduation from high school within 4 years, the student received a 
local diploma. The local diploma had less rigorous requirements than the New York State Regents diploma and 
was phased out subsequent to the study period. 

New York State Regents diploma 
granted 

A binary measure indicating whether, upon graduation from high school within 4 years, the student received a 
New York State Regents diploma, the standard diploma granted in New York State. 

Advanced Regents diploma granted 

A binary measure indicating whether, upon graduation from high school within 4 years, the student received an 
Advanced Regents diploma, which has more rigorous requirements than the standard New York State Regents 
diploma. 

General literacy achievement 

College readiness in English 

A binary measure indicating whether the student obtained a score of 75 or higher on the English Regents exam, 
the threshold whereby students are exempted from remedial English coursework at the City University of New 
York (CUNY). 

Math achievement 

College readiness in math 

A binary measure indicating whether the student obtained a score of 75 or higher on the Math A Regents exam, 
the threshold whereby students were exempted from remedial math coursework at the CUNY until 2012. 


Table Notes: In addition to the 4-year graduation rates, the 5-year graduation rates were also assessed for cohorts 1 and 2 in the analysis of the effect of attending an SSC (see 
the main body of the study report). In the analysis of the effect of attending an SSC, the authors also presented effects on 4-year graduation rates for subgroups defined according 
to the following characteristics: cohort (1-3); eighth grade proficiency level in reading and math (level 1-4); eligibility for free or reduced-price lunch; race/ethnicity, by gender 
(Black male, Black female, Hispanic male, Hispanic female, other male, other female); whether the student was “known” to the SSC; the student's ranking of the SSC in his or her 
enrollment preferences (first, second, or other); and whether the student was eligible for special education or English language learner services. The authors did not present 5-year 
graduation rates or subgroup effects for the intent-to-treat analysis, which is presented in Appendix B of the study report, and therefore, these results are not included in this WWC 
report. 
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Appendix C: Study findings for each domain 





Mean 

(standard deviation) 


WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Completing school 

Graduated from high school 

SSC lottery 
participants 

12,130 

students 

0.70 

(na) 

0.64 

(na) 

0.06 

0.16 

+7 

<0.001 

Domain average for school attendance 





0.16 

+7 

Statistically 

significant 

General literacy achievement 

College readiness In English 

SSC lottery 
participants 

12,130 

students 

0.40 

(na) 

0.35 

(na) 

0.04 

0.11 

+4 

< 0.001 

Domain average for general literacy achievement 




0.11 

+4 

Statistically 

significant 

Math achievement 

College readiness in math 

SSC lottery 
participants 

12,130 

students 

0.24 

(na) 

0.25 

(na) 

0.00 

-0.01 

0 

0.790 

Domain average for math achievement 





-0.01 

0 

Not 

statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the average change expected for all students 
who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average student's percentile rank that can be expected if the student is given the intervention. The statistical significance of the study's domain average was deter- 
mined by the WWC. na = not applicable. 

Study Notes: No corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-values presented here were reported in the 
original study. The intervention group means are unadjusted. The comparison group means reported by the author represent the intervention group mean minus the intervention 
effect based on an Ordinary Least Squares (OLS) regression which controlled for students' SSC lottery and their baseline reading and math scores. The results for specific types of 
diplomas received by students is presented in Appendix D. 

This study is characterized as having a statistically significant positive effect on school attendance and general literacy achievement because the effect in each domain is positive 
and statistically significant. This study is characterized as having indeterminate effect on math achievement because the effect is neither statistically significant nor substantively 
important. For more information, please refer to the WWC Standards and Procedures Handbook, version 2.1 , page 96. 
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Appendix D: Supplemental findings by domain 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Mean 

(standard deviation) 


WWC calculations 

p-value 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

Completing school 

Local diploma granted 

SSC lottery 

12,130 

0.17 

0.16 

0.02 

0.07 

+3 

0.061 


participants 

students 

(na) 

(na) 





New York State Regents 

SSC lottery 

12,130 

0.45 

0.41 

0.04 

0.09 

+4 

0.002 

diploma granted 

participants 

students 

(na) 

(na) 





Advanced Regents diploma 

SSC lottery 

12,130 

0.08 

0.07 

0.01 

0.06 

+3 

0.172 

granted 

participants 

students 

(na) 

(na) 






Table Notes: The supplemental findings presented in this table are additional findings that do not factor into the determination of the evidence rating. For mean difference, effect 
size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the comparison group. The effect size is a 
standardized measure of the effect of an intervention on student outcomes, representing the average change expected for all students who are given the intervention (measured in 
standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an average student’s percentile rank 
that can be expected if the student is given the intervention, na = not applicable. 

Study Notes: A correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. The p-values presented 
here were reported in the original study. 

The intervention group means are unadjusted. The comparison group means reported by the author represent the intervention group mean minus the intervention effect based on an 
Ordinary Least Squares (OLS) regression which controlled for students’ SSC lottery and their baseline reading and math scores. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the single study review protocol, version 2.0. A quick review of this study was released in 
January 2012, and this report is the follow-up review that replaces that initial assessment. 

2 This single study review reports the results of the intent-to-treat (ITT) analysis of each student’s first SSC lottery, found in Appendix B 
of the study report. This analysis measures the effect of the offer to attend an oversubscribed SSC. The main body of the report pres- 
ents the effect of actually attending an SSC, rather than receiving an offer to attend an SSC. In studies that report an ITT effect (e.g., 
the offer of admission) in conjunction with a compiler average causal effect (e.g., attendance in an SSC), WWC standards require that 
the review should be based only on the ITT effect, because fewer assumptions are needed for the ITT estimate to provide an internally 
valid test of program impact. As a result, the results of the analyses focusing on attending SSCs are not included in this single-study 
review. All analyses included lottery fixed effects in order to ensure that all intervention-comparison evaluations are based on the 
experimental within-school evaluations. 

3 There was one outcome included in the study that is not described in this WWC report, as well as additional analyses of the effect of 
attending an SSC for a number of subgroups. See the table notes in Appendix B for more information. Because the type of diploma 
outcome is effectively a “subtest” of the broader high school graduation rate outcome, these results are presented in Appendix D. 

4 All students with missing baseline data were excluded from the baseline sample in the authors’ analyses. Students who entered the 
lottery from a private or parochial school were excluded from the sample, regardless of whether they won or lost the lottery, because 
they could not be tracked following the lottery. In addition to these exclusions based on pre-random assignment characteristics, some 
full lotteries (including both lottery winners and losers) were excluded from the sample. This occurred (a) when the aforementioned 
exclusions resulted in no students remaining in either the intervention or comparison group; (b) when the lottery school did not open 
as planned; or (c) when the percentage of lottery winners and losers enrolling in the SSC was equal. These full lottery exclusions limit 
the external validity of the study findings but do not compromise the internal validity of the experiment. 

5 The demographic characteristics reported in Appendix B were calculated by the WWC by pooling the prevalence rates reported by 
condition in the study. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2014, February). WWC 
review of the report: Sustained progress: New findings about the effectiveness and operation of small public 
high schools of choice in New York City. Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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